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Abstract 

We  consider  the  robust  optimal  control  of  a  law  of  large  numbers 
approximation  of  a  stochastic  network.  The  robust  control  problem  is 
formulated  as  a  differential  game,  with  one  player  choosing  the  policies 
that  determine  service  and  routing  assignments,  and  the  other  choosing 
quantities  such  as  the  arrival  and  service  rates,  subject  to  constraints. 
The  cost  to  be  minimized  by  the  first  player  and  maximized  by  the 
second  is  the  time  till  the  origin  is  reached.  An  explicit  formula  is  given 
for  the  value  function,  and  some  of  its  basic  properties  are  studied. 


1  Introduction 

This  paper  considers  the  problem  of  robust  service  and  routing  control  for 
a  network  of  servers.  Consider  such  a  network,  and  assume  that  at  each 
station  there  are  a  finite  number  of  distinct  customer  classes,  each  with  its 
own  buffer.  In  this  paper  we  will  work  directly  with  what  is  sometimes  called 
a  “fluid”  model  for  the  network  [16].  Models  of  this  sort  are  usually  obtained 
as  law  of  large  numbers  approximations  to  more  detailed  models  [5,  15], 
and  are  particularly  appealing  because  in  many  cases  related  optimization 
problems  admit  closed  form  solutions  [20,  21,  11]. 

Another  feature  of  the  networks  we  consider  is  model  uncertainty,  such 
as  uncertainty  in  the  arrival  and  service  rates.  To  deal  with  model  un¬ 
certainty  we  adapt  the  differential  game  formulation  of  robust  control  for 
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unconstrained  nonlinear  systems  [14],  Thus  we  consider  a  network  where 
there  are  two  players.  One  player  in  the  game  will  represent  the  “true” 
control  (e.g.,  service  assignments  and  routing  decisions).  The  other  player 
represents  the  uncertain  or  poorly  modeled  aspects  of  the  system  (e.g.,  ar¬ 
rival  and  service  rates).  In  keeping  with  existing  convention,  we  will  refer  to 
this  latter  control  as  “nature.”  The  two  players  are  antagonistic,  with  the 
first  player  attempting  to  maintain  good  system  performance. 

Differential  game  formulations  provide  a  powerful  tool  for  the  design  of 
robust  controls  [3,  14].  In  many  situations  knowledge  of  the  true  system  is 
limited.  System  parameters  (e.g.,  arrival  rates)  may  drift  with  time,  and 
statistical  properties  (e.g.,  correlations)  may  also  be  unstable.  There  may 
be  aspects  of  the  system  that  are  left  unmodeled,  either  because  they  cannot 
be  estimated  in  any  reliable  way,  or  because  they  lead  to  a  model  that  is 
too  complicated  to  be  useful.  This  is  a  common  occurence  in  stochastic 
networks,  where  the  network  to  be  controlled  is  often  a  sub-network  of  some 
larger  system,  and  “full  state  information”  is  simply  not  available  to  the 
controller  of  the  sub-network. 

In  situations  like  these  the  use  of  a  single  “nominal”  model  can  be  prob¬ 
lematic.  For  example,  just  as  in  the  case  of  unconstrained  systems  one  can 
construct  examples  where  controls  that  are  optimal  in  some  sense  for  the 
nominal  model  perform  poorly  when  the  model  is  perturbed  even  slightly. 
A  differential  game  formulation  allows  one  to  contruct  controls  that  perform 
uniformly  well  over  a  class  of  perturbations  of  the  nominal  model,  with  each 
choice  of  nature’s  control  corresponding  to  a  different  perturbation  of  the 
design  model.  It  is,  of  course,  this  insensitivity  to  model  perturbations  that 
warrants  the  term  “robust”  control.  Variations  of  different  kinds  can  be 
accomodated  through  the  choice  of  the  cost  structure,  and  one  can  carefully 
balance  the  pursuit  of  optimality  with  respect  to  a  nominal  model  against 
the  need  to  provide  good  performance  for  a  range  of  models.  The  main 
result  of  this  paper  is  the  explicit  solution  to  a  robust  control  problem  for  a 
network.  By  explicit  what  we  mean  is  that  the  value  function  can  be  repre¬ 
sented  in  terms  of  a  finite  dimensional  optimization  problem,  and  that  from 
this  value  function  one  can  obtain  controls  with  specific  robust  properties. 

In  formulating  the  differential  game  special  attention  must  be  paid  to  the 
cost  applied  to  nature’s  control,  since  this  determines  the  degree  to  which 
model  perturbations  are  allowed.  Within  the  realm  of  “fluid”  models  there 
are  at  least  two  types  of  cost  structures  that  are  natural.  One  is  a  cost 
that  simply  imposes  a  constraint  on  the  model  parameters.  We  will  refer 
to  this  as  the  case  of  a  “hard  constraint.”  An  alternative  is  to  make  nature 
pay  an  increasing  cost  for  perturbations  away  from  the  nominal  model,  and 
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we  will  refer  to  this  type  of  cost  as  corresponding  to  “soft  constraints.” 
Hard  constraints  turn  out  to  be  mathematically  simpler  for  fluid  models 
of  networks,  even  though  the  reverse  seems  to  be  true  for  unconstrained 
systems.  In  the  present  paper  we  will  focus  on  the  case  of  hard  constraints, 
and  defer  the  case  of  soft  constraints  to  future  work. 

Besides  the  cost  that  nature  must  pay,  one  must  also  specify  the  cost  that 
the  true  control  faces.  In  this  paper  the  cost  we  consider  is  the  time  to  move 
the  state  of  the  system  from  an  arbitrary  position  to  zero  (i.e.,  all  queues 
empty).  The  true  control  will  try  to  minimize  this  time,  while  the  opposing 
control  will  attempt  to  delay  it  as  much  as  possible.  This  cost  seems  to 
be  a  natural  analogue,  in  the  setting  of  constrained  systems,  of  the  familiar 
quadratic  cost  for  unconstrained  systems.  In  particular,  it  leads  to  controls 
with  optimal  robust  stability  (in  addition  to  optimal  robust  performance), 
and  it  also  allows  for  a  fairly  explicit  closed  form  solution. 

An  outline  of  the  paper  is  as  follows.  In  Section  2  we  give  a  precise 
formulation  of  the  game  problem,  and  show  through  examples  how  various 
systems  can  be  put  into  this  framework.  Sections  3  includes  the  main  result 
of  the  paper,  which  is  a  finite  dimensional  max/min  representation  for  the 
value  function  of  the  game  introduced  in  Section  2.  Qualitative  properties 
of  the  value  function  (convexity,  differentiability,  etc.)  are  also  discussed 
in  Section  3.  The  proof  of  this  representation  is  given  in  Section  4.  The 
concluding  Section  5  formally  discusses  how  an  optimal  true  control  can  be 
constructed  in  feedback  form.  A  proof  of  the  existence  of  value  for  the  games 
we  consider  is  given  in  an  Appendix. 

2  Formulation  of  the  Control  Problem 

In  this  section  we  formulate  the  robust  control  problem  as  a  constrained 
deterministic  differential  game.  As  discussed  in  the  introduction,  the  model 
we  use  can  be  viewed  as  a  law  of  large  numbers  approximation  to  a  more 
detailed  stochastic  model.  This  connection  will  be  used  for  interpretive 
purposes  throughout  the  section.  The  state  space  of  the  process  is  1R+ ,  and 
one  can  interpret  each  of  the  components  as  a  queue  length  associated  with 
a  specific  customer  class. 

The  formulation  of  the  model  involves  two  collections  of  iV— dimensional 
vectors.  The  first  are  the  directions  of  constraint,  which  we  designate  by 
{di,i  =  1,  These  vectors  are  used  to  define  the  Skorokhod  or  reflec¬ 

tion  map,  which  properly  corrects  the  dynamics  of  the  model  when  one  or 
more  components  of  the  state  are  zero  (i.e.,  one  or  more  customer  classes  are 
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empty).  The  second  collection  is  designated  {vjk,j  =  1,  •••,  J,k  =  1, K}, 
and  is  used  to  define  the  dynamics  of  the  system  away  from  the  boundary. 
Nature’s  control  takes  values  in  a  compact  convex  set  A  C  IR^,  and  the  in¬ 
dex  j  £  {1, ...,  J}  corresponds  to  one  of  the  possible  “pure”  service/routing 
configurations  the  true  controller  can  select  (illustrative  examples  will  be 
given  below) .  If  nature  chooses  the  control  a  £  A  and  the  true  control  is  the 
pure  configuration  j,  then  the  quantity  J2i^=i  ctkVjk  characterizes  the  (law 
of  large  numbers)  evolution  of  the  network  when  the  state  of  the  network 
is  away  from  OIR:^.  More  general  service/routing  policies  can  be  obtained 
by  considering  convex  combinations  of  the  pure  controls,  in  which  case  the 
velocity  of  the  system  is  given  by 


J  K 

F'{p,a)  =  '^'^PjakVjk, 
i=l k=l 


where 


P  =  {pi,-,Pj)  G  S 


X  £  1R‘'  :  X,-  >  0,j  =  1, 


J 

i=i 


Xn  =  1 


and  pj  is  the  fraction  of  time  allocated  to  the  pure  configuration  j. 

We  will  assume  the  following  condition  on  the  directions  of  constraint. 
The  condition  is  by  now  classical  in  the  study  of  approximations  to  queueing 
networks,  and  is  called  the  Harrison- Reiman  condition  in  [10].  It  was  first 
used  in  [13].  Although  the  Harrison- Reiman  condition  is  usually  associated 
with  single  class  networks,  it  also  defines  the  proper  Skorokhod  Problem  for 
many  formulations  of  controlled  multiclass  networks  as  well.  Note  that  the 
condition  is  the  original  Harrison-Reiman  condition,  and  not  the  generaliza¬ 
tion  that  is  also  studied  in  [10]. 


Condition  2.1  For  each  i  £  {l,...,iV} 

{di)i  =  1,  and  {di)j  <  0  for  j  ^  i. 

Let  D  be  the  matrix  whose  ith  column  is  di.  Then  the  spectral  radius  of 
I  —  D  is  less  than  1. 


The  following  simple  examples  illustrate  the  role  these  different  quan¬ 
tities  play.  The  tth  unit  basis  vector  is  denoted  by  Cj.  Some  of  the  most 
difficult  aspects  in  the  control  of  networks  are  due  to  feedback  and  the  in¬ 
teractions  between  different  servers.  ^From  this  perspective,  the  first  two 
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examples  are  too  simple  to  be  of  great  interest.  Also,  it  should  be  noted 
that  the  game  formulation  we  consider  in  this  paper  only  allows  routing  at 
the  “fringes”  of  the  network,  and  not  between  nodes.  We  hope  to  consider 
the  more  general  routing  problem  in  future  work. 

Example  1.  The  first  example  is  a  simple  routing  control  problem.  The 
rate  of  arrivals  to  the  router  is  A(t),  and  the  service  rates  of  the  two  servers 
are  and  respectively.  The  system  is  illustrated  in  Figure  1. 
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Figure  1:  Simple  routing  control 

This  model  is  put  into  the  framework  described  above  by  setting 

ai  =  m  upi  =  -ei  V2,i  =  -ei 

02  =  M2  ^^1,2  =  —62  V2,2  =  —62  • 

as  =  A  Ui^S  =  Cl  V2,3  =  62 

The  choice  of  A  determines  the  uncertainties  and  perturbations  against 
which  the  optimal  true  control  will  be  robust.  For  example,  if  the  nomi¬ 
nal  service  and  arrival  rates  are  A  =  1  and  fii  =  1,  i  =  1,2,  and  if  the  service 
rates  are  well  modeled  and  the  arrival  rate  less  so,  then  one  might  consider 
a  set  of  the  form 

A  =  [Af, /if ]  X  [m^, X  [A^, F]  =  [.9, 1.1]  X  [.9, 1.1]  X  [.5, 1.5]. 
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This  model  is  very  simple,  and  perhaps  too  simple  to  capture  any  “proba¬ 
bilistic”  intuition.  For  example,  there  is  no  constraint  on  combinations  of 

and  ^2-  ^From  a  probabilistic  perspective  one  might  imagine  that  it  is 
less  likely  that  both  of  these  parameters  would  equal  their  minimum  value 
at  the  same  time.  The  introduction  of  a  constraint  to  account  for  this  would 
lead  to  a  set  A  with  a  “curved”  boundary. 

One  might  also  wish  to  consider  an  increasing  family  of  sets  A{c)  indexed 
by  c  G  [0,oo),  and  with  .4(0)  just  the  nominal  model.  The  largest  c  such 
that  a  certain  robust  performance  measure  can  be  met  (e.g.,  finiteness  of 
the  value  function)  is  an  important  quantity.  In  particular,  it  characterizes 
the  control  that  is  most  robust,  where  the  sense  of  robustness  is  determined 
by  the  shape  of  A  and  the  relative  uncertainty  it  assigns  to  different  aspects 
of  the  network. 

We  return  our  consideration  to  the  particular  example  of  Figure  1.  If 
a  service  is  attempted  at  server  1  and  the  queue  is  empty  then  the  proper 
compensating  action  is  simply  to  return  queue  1  to  the  level  zero.  As  a 
consequence,  the  direction  of  constraint  for  the  corresponding  face  is  just 
di  =  (1,0).  A  corresponding  remark  applies  to  queue  2. 

Example  2.  Here  we  consider  a  simple  service  control  problem.  The  system 
is  illustrated  in  Figure  2. 
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Figure  2:  Simple  service  control 


The  Skorokhod  Problem  for  this  is  the  same  as  that  for  Example  1.  This 
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model  is  put  into  the  standard  framework  by  setting 


ai  —  111  vi^i  —  — ei 
02  =  M2  ^^1,2  =  0 
03  =  Ai  ui^4  =  ei 
04  =  A2  Ui^4  =  62 


t>2,l  =  0 
■^2,2  =  —62 
■62,4  =  ei 
62,4  =  62 


Example  3.  This  example  considers  a  network  of  servers,  and  as  a  con¬ 
sequence  the  associated  Skorokhod  Map  is  more  involved.  The  network  is 
illustrated  in  Figure  3.  Since  there  are  6  customer  classes  the  domain  is  IR^. 
Suppose  the  service  rate  for  class  i  is  the  arrival  rate  is  \{t). 


Figure  3:  Network  model 


An  example  of  a  pure  configuration  (labeled  say  j),  is  to  route  to  class 
1  and  serve  classes  3,  2  and  5.  If  we  let  (oi, og,  07)  =  (mi,  •••,  M6)  then 
the  vectors  Vjk  for  /c  =  2,  3, 5,  7  are 

Vj2  =  (-62  +  66),6j3  =  (-63  +  64),Uj5  =  (-65  +  e3),Vj7  =  64, 

while  Vji  =  Uj4  =  VjQ  =  0.  The  velocity  of  the  network  under  this  configu¬ 
ration  is  M26j2  +  M36j3  +  M56j5  +  At>j7. 

If  a  service  is  attempted  for  say  customer  class  i  =  3  and  the  queue 
is  empty,  then  queue  3  must  be  returned  to  zero  and  in  addition  queue  4 
must  be  reduced  by  the  same  amount.  Consequently,  the  proper  direction 
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of  constraint  for  face  i  =  3  is  da  =  (es  —  64).  Analogous  considerations  can 
be  used  to  identify  all  other  directions  of  constraint. 


Example  4.  In  some  problems  there  is  randomized  (uncontrolled)  routing. 
For  example,  after  service  a  fraction  9j  of  the  class  i  customers  may  become 
class  j  customers,  and  a  fraction  Oq  =  1  —  of  th®  customers  could 

leave  the  system.  Let  {di)j  =  —6j  \i  j  ^  i  and  {di)i  =  1.  Then  the  direction 
of  constraint  is  di  on  the  face  {x  G  IR;^  :  =  0},  and  the  reason  is  the  same 

as  in  the  last  case:  compensating  for  a  “fictitous”  service  of  a  customer 
of  class  i  requires  a  boost  to  coordinate  i  and  a  corresponding  decrease  in 
coordinate  j  with  constant  of  proportionality  9j  [18]. 


To  formulate  the  robust  control  problem  we  must  specify  the  dynamics. 
Let 

C'+([0,oo)  :  1R^)  =  {V^  G  <^^([0,00)  :  IR^)  :  V’(O)  G  IR^"}, 

where  C'([0,oo)  :  IR'^)  is  the  usual  space  of  continuous  functions  with  the 
sup  norm  metric,  and  suppose  that  a  set  of  vectors  that  satisfy  Condition 
2.1  is  given.  For  each  point  x  on  the  boundary  of  IR;^  let 


d{x) 


\  ^  ^  Ojidi  .  Qi  ^  0, 

^  ^  O^idi 

where  I{x)  =  {i  :  =  0}.  The  Skorokhod  Problem  assigns  to  every  path 

■0  G  00)  :  IR'^)  a  path  0  that  starts  at  0(0)  =  0(0),  but  is  constrained 

to  IR:^  as  follows.  If  0  is  in  the  interior  of  IR:^  then  the  evolution  of  0  mimics 
that  of  0,  in  that  the  increments  of  the  two  functions  are  the  same  until  0 
hits  the  boundary  of  IR:]^.  When  0  is  on  the  boundary  a  constraining  “force” 
is  applied  to  keep  0  in  the  domain,  and  this  force  can  only  be  applied  in  one 
of  the  directions  d{(j){t)),  and  only  for  t  such  that  0(t)  is  on  the  boundary. 
The  precise  definition  is  as  follows.  For  i]  G  (^([OjOo)  :  IR'^)  and  t  G  [0,oo) 
we  let  \ri\{t)  denote  the  total  variation  of  r]  on  [0,t]  with  respect  to  the 
Euclidean  norm  on  IR'^. 


Definition  2.1  Let  0  G  C'+([0,oo)  :  IR'^)  be  given.  Then  {(f), rj)  solves  the 
SP  for  0  (with  respect  to  IR:^  and  di,i  =  1, ...,  N)  if  0(0)  =  0(0),  and  if  for 
all  t  G  [0, 00) 

1.  0(0  =  ^(^t)  +  r]{t), 

2.  0(0  G  G, 
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3.  \r]\{t)  <  oo, 

4-  \ri\(.t)  =  /[o,t] 

5.  There  exists  a  Borel  measurable  function  7  :  [0,  00)  ^  IR^  such  that 
d\ri\-almost  everywhere  7(t)  G  d{f{t)),  and  such  that 

h{t)=  j  7('S)t^l??l(s)- 

Note  that  rj  changes  only  when  f  is  on  the  boundary,  and  only  in  the  direc¬ 
tions  d{(p). 

Under  Condition  2.1  the  Skorokhod  Problem  has  a  solution  for  all  if  G 
C'+([0,oo)  :  IR'^).  In  addition,  the  mapping  if  ^  (p  is  Lipschitz  continuous 
[8,  13], 

We  next  define  a  constrained  ordinary  differential  equation.  As  is  proved 
in  [8],  one  can  define  a  projection  vr  :  IR'^  ^  IR:^  that  is  consistent  with  the 
constraint  directions  {di,i  =  1,...,A^},  in  that  7r(x)  =  x  if  x  G  IR;^,  and  if 
X  ^  IR:^  then  7r(x)  —  x  =  ar,  where  a  >  0,  7r(x)  G  cllR:^,  and  r  G  d{7r{x)). 
Figure  4  illustrates  the  projection  for  a  two  dimensional  problem. 


Figure  4:  The  discrete  projection 


With  this  projection  given,  we  can  define  for  each  point  x  G  cllR:^  and  each 
V  G  IR'^  the  projected  velocity 


7r(x,  v)  =  lim 
AiO 


7r(x  -|-  Ax)  —  7r(x) 

A  ■ 


(1) 
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For  details  on  why  this  limit  is  always  well  defined  and  further  properties 
of  the  projected  velocity  we  refer  to  [6,  Section  3  and  Lemma  3.8]  and  [7]. 
The  dynamical  model  for  the  game  we  consider  is  then  given  by 

^{t)  =  7r{(j){t),  F{p{t),a{t))),  (2) 

where 

J 

F{p{t),a{t))  =  ^pj{t)ak{t)vjk,  (3) 

i=i 

and  for  all  t  G  [0,  oo)  the  true  control  p{t)  takes  values  in  the  set  S  and 
nature’s  control  a{t)  takes  values  in  the  set  A.  According  to  the  Skorokhod 
Problem,  the  velocity  F(p,  a)  governs  the  evolution  of  the  network  when  all 
states  are  positive.  When  one  or  more  states  are  negative,  the  projection 
of  the  velocity  provides  the  proper  correction  to  the  dynamics  due  to  non¬ 
negativity  constraints. 

An  absolutely  continuous  function  (j)  :  [0,  oo)  ^  IR:]^  is  a  solution  to 
(2)  if  the  equation  is  satisfied  in  an  a.e.  sense  in  t.  By  using  the  regular¬ 
ity  properties  of  the  associated  Skorokhod  Map,  one  can  prove  that  all  the 
standard  qualitative  properties  (existence  and  uniqueness  of  solutions,  sta¬ 
bility  with  respect  to  perturbations,  etc.)  hold  [8].  In  fact,  because  of  the 
particularly  simple  nature  of  the  right  hand  side  (i.e.,  7r{(j){t),  P{t))  rather 
than  b{4>{t))  +  P{t))  for  some  function  b),  one  can  show  that  (p  solves 

(2)  if  and  only  if  p  is  the  image  of  ■0(t)  =  /q  F{p{s),  a{s))ds  -|-  x  under  the 
Skorokhod  Map,  in  which  case  all  such  issues  become  trivial  [8]. 

The  ODE  (2)  defines  the  dynamics  for  the  game  that  we  will  consider. 
The  cost  we  consider  is  the  time  for  the  state  to  reach  the  origin,  which  the 
true  control  will  attempt  to  minimize  and  which  nature  will  try  to  prolong. 
As  usual  in  differential  games,  one  must  deal  with  the  issue  of  which  player 
has  the  “information  advantage”  [12].  For  the  problems  we  consider  it  will 
always  turn  out  that  the  game  has  value,  and  so  the  value  function  will  be 
the  same  regardless  of  who  has  the  information  advantage. 

We  use  the  standard  Elliot-Kalton  formulation  of  the  game.  Define  the 
spaces  of  (open  loop)  controls 

N  =  {p  :  [0,  oo)  ^  S  :  p  is  measurable} 


and 

M  =  {a  :  [0,  oo)  ^  M  :  a  is  measurable}. 

We  identify  any  two  controls  that  are  equal  almost  everywhere.  Given  x  G 
IR:]]^,  the  dynamics  of  the  game  are  given  by  (2)  and  (3).  Associated  with 
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these  dynamics  is  the  cost 


Cx{p,a)  =  Tx, 

where  Tx  =  inf{t  >  0  :  4>{t)  =  0}.  A  mapping  6  :  N  ^  M  is  said  to  be  a 
strategy  for  the  maximizing  player  if  for  each  s  >  0  and  p,  p  £  N 

p{t)  =  p{t)  for  a.e.  0  <  f  <  s 


implies 

9[p]{f)  =  9[p\{t)  for  a.e.  0  <t  <  s. 

A  strategy  for  the  minimizing  player,  which  will  be  denoted  by  6,  is  defined 
in  an  analogons  manner.  We  denote  by  0  the  set  of  all  maximizing  strategies 
and  by  A  the  set  of  all  minimizing  ones.  The  lower  value  of  the  game  and 
the  upper  value  of  the  game  are  defined  by 


V  (x)  =  inf  snp  Cx{h[a],a)  (4) 


and 


l/+(x)  =  snp  M  Cx{p,9[p]) 

0(zQP&N 


(5) 


respectively.  If  V~{x)  =  V^{x),  then  the  game  is  said  to  have  value. 

Let  V  :  IR'^  ^  IR.  For  points  x  £  IR'^  and  directions  w  £  IR'^  for  which 
the  limit  exists,  we  let  DwV{x)  denote  the  directional  derivative  in  direction 
w  at  x: 


DyjV  (x)  =  lim 
aj.0 


V{x  +  aw)  —  V  (x) 
a 


We  say  that  V  is  radially  linear  if  V{ax)  =  aV{x)  for  all  x  £  and 
a  £  [0,  oo). 


3  Representation  for  the  Value  Function 

For  V^{x)  and  V~{x)  to  be  finite  we  will  need  some  conditions.  Define  the 
convex  cone 

C  =  <  -  ^  aidi  :  Oj  >  0,  f  =  1, ...,  N 
I  i=l 

which  is  the  negative  of  the  cone  of  constraint  directions  that  are  allowed  at 
the  origin.  As  observed  in  [6],  this  cone  can  be  used  to  characterize  stability 
conditions  for  (2). 
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The  following  formula  gives  an  explicit  representation  for  the  value  of 
the  game  defined  in  the  last  section.  The  precise  statement  is  given  at 
the  end  of  the  section,  and  the  proof  is  given  in  Section  4.  Recall  that 
P{P^  a)  =  E/=i  Ef=i  PjOikVjk.  Then  set 

W (x)  =  sup  inf  inf  {a  :  x  +  a'F{p,  a)  G  C}  .  (6) 

We  will  also  make  use  of 

Wa{x)  =  inf  inf  {a  :  x  +  <7F{p,  a)  G  C}  .  (7) 

pe5 

The  following  condition  is  necessary  and  sufficient  for  W  (x)  to  be  finite 
for  all  X  G  IR^.  Let  C°  denote  the  interior  of  C. 

Condition  3.1  For  each  a  ^  A  there  exists  p  £  S  such  that 

F{p,a)  G  C°. 

It  follows  directly  from  the  definition  of  Wa{x)  that  under  this  condition 
Wa{x)  <  oo  for  all  x  G  IR'^.  Since  A  is  compact,  an  open  covering  argument 
can  be  used  to  prove  that  W{x)  <  oo  for  all  x  G  IR'^. 

In  order  to  motivate  the  representation  (6),  we  first  consider  (7).  In  this 
case  there  is  just  “true”  control  for  a  fixed  set  of  arrival  and  service  rates. 
It  turns  out  that  equals  the  minimum  time  for  a  control  problem  that 
uses  the  dynamics  defined  by  the  Skorokhod  Problem  and  stops  when  the 
origin  is  reached.  However,  from  the  formula  for  Wa  it  is  clear  that  Wa 
equals  the  solution  to  the  minimum  time  problem  with  the  much  simpler 
dynamics  (/>(t)  =  F(p(t),  a)  and  the  stopping  set  C.  Away  from  the  boundary 
(IIR:^  these  two  different  minimum  time  problems  should  satisfy  the  same 
Hamilton-Jacobi  Bellman  equation. 
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Figure  5:  Level  sets  of  Wa'-  Classical  boundary  conditions 


Owing  to  the  constraining  dynamics,  the  first  minimum  time  problem  should 
satisfy  a  Neumann  boundary  condition  {DWa{x) ,  di)  =  0  for  f  G  I{x)  on 
91R:^\{0}  (in  the  viscosity  sense).  It  turns  out  (under  Condition  2.1)  that 
the  shape  of  the  stopping  set  C  in  the  second  minimum  time  problem  pro¬ 
duces  a  function  whose  gradient  satisfies  this  boundary  condition,  and  so  by 
uniqueness  one  would  expect  the  two  minimum  time  problems  to  coincide 
on  IR:^  [1].  Figure  5  illustrates  the  situation  for  a  particular  two  dimensional 
problem  with  no  control  (so  that  F{p,a)  =  u  is  a  constant).  The  dotted 
lines  indicate  level  curves  of  Wa,  and  since  these  level  curves  are  parallel  to 
di  for  X  near  {x  G  (1R^)\{0}  :  Xj  =  0},  the  boundary  conditions  hold,  even 
in  a  classical  sense. 

The  situation  is  not  always  so  simple,  as  indicated  by  Figure  6. 


Figure  6:  Level  sets  of  Wa'-  Without  classical  boundary  conditions 


Here  only  one  boundary  condition  holds  in  the  classical  sense,  and  this  is  due 
to  the  fact  that  at  the  other  boundary  v  points  into  the  interior  and  away 
from  this  boundary.  One  of  the  important  properties  of  viscosity  solutions 
is  that  they  allow  such  relaxations. 
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The  remarkable  fact  is  that  an  analogous  representation  continues  to 
hold  even  in  the  game  problem,  with  simply  an  additional  supremization  on 
a  £  A.  It  should  be  noted  that  even  though  the  game  has  value,  one  cannot 
permute  the  infpg^  and  supQ,g_4  in  (6). 

In  the  rest  of  this  section  we  will  prove  qualitative  properties  of  W  that 
are  needed  for  the  proof  that  W  is  the  value  of  the  game. 

Theorem  3.1  Assume  that  Conditions  3.1  and  2.1  are  satisfied  and  define 
Wa(x)  for  X  £  IR'^  and  a  £  A  by  (7).  The  following  conelusions  hold. 

1.  Wa  is  finite  and  radially  linear  on 

2.  For  each  x  £  IR'^  the  infimum  in  (7)  is  achieved  at  some  probability 
vector  p. 

3.  Wa  is  convex  on  IR'^. 

4.  Wa{x)  >0  forx£  IR^\{0}. 

Proof:  Under  Condition  3.1  it  is  obvious  that  the  cone  C  can  be  reached 
from  any  starting  point  x,  and  so  Waix)  <  oo,  while  radially  linearity  is 
an  immediate  consequence  of  the  definition  of  Waix).  It  follows  from  the 
compactness  of  S  that  the  infimum  is  achieved  in  the  definition  of  Waix). 
Thus  the  proofs  of  parts  1  and  2  are  complete. 

To  prove  property  3  we  first  consider  points  xi  and  X2  such  that  Waixi)  = 
Waix2)  A  0.  Let  c  denote  the  common  value,  and  let  and  p^  denote 
minimizing  probability  vectors  in  the  expression  that  defines  Waixi)  and 
Waix2),  respectively.  Thus  Xi  +  cF(/9*,a)  G  dC  for  i  =  1,2.  For  s  £  [0,1], 
the  convexity  of  C  implies 

sxi  +  (1  —  s)x2  +  scFip^,  a)  +  (1  —  s)cFip‘^,  a) 

=  isxi  +  (1  -  s)x2)  +  cFisp^  +  (1  -  s)p‘^,  a)  £  C. 

Since  sp^  +  il  —  s)p^  £  S,  it  follows  that 

Waisxi  +  (1  -  s)x2)  <c  =  sWaixi)  +  (1  “  s)Waix2). 

We  next  consider  the  case  of  any  points  xi  and  X2  such  that  Waixi)  A  0 
and  Waix2)  A  0-  Let 


C  =  sWaixi)  +  (1  -  s)Waix2). 
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Since  Wa  is  radially  homogeneous, 


Wa{sxi  +  (1  -  s)x2) 

W^{xi) 


= 


= 


Wa(xi) 

sWa{xi) 


,  ^Wa{x2) 

+  (1  -  s)T7r^^X2 


Xi 


< 


sWa{xi) 


Wa(xi) 

Xi 

Wa{xi)'' 


+ 


w^{x2y 

(1  -  s)Wa{x2) 


X2 


+ 


C  Wa{x2) 

(1  -  s)Wa{x2) 


-w„ 


X2 


Wa(x2) 


=  sWc{xi)  +  {I  -  s)Wa{x2)- 


The  case  where  Wa{xi)  or  1Tq;(x2)  equals  zero  is  similar  and  omitted. 

Finally  we  must  prove  property  4.  The  Harrison- Reiman  condition  im¬ 
plies  what  is  called  the  completely-S  condition  ([17,  4]),  which  requires  the 
existence  of  a  vector  v  G  IR'^  satisfying  Vi  >  0,i  =  and  {v,j)  >  0 

for  all  7  G  d{0).  Hence  if  y  G  C\{0}  then  (u,y)  <  0,  and  so  y  ^  IR:^.  This 
shows  that 

CnlR^  =  {0}, 

and  therefore  1Tq:(x)  >  0  for  all  x  G  1R:^\{0}.  ■ 


Theorem  3.2  Assume  that  Conditions  3.1  and  2.1  are  satisfied  and  define 
W(x)  for  X  G  IR'^  by  (6).  The  following  conelusions  hold. 

1.  W  is  finite  and  radially  linear  on  IR'^. 

2.  W  is  eonvex  on  IR'^. 

3.  W(x)  >0  for  xe  1R^\{0}. 

Proof:  R  follows  from  Condition  3.1  that  for  each  x  G  IR'^  Wa(x)  is  bounded 
uniformly  in  a  G  .A.  All  the  claims  then  follow  from  the  preceding  theorem 
and  W{x)  =  sup„g_4  1Tq,(x).  ■ 


Remark.  Since  W  is  convex,  directional  derivatives  exist  at  all  points  and 
for  all  directions. 


Theorem  3.3  Assume  that  Conditions  2.1  and  3.1  are  satisfied  and  define 
W {x),V^ {x)  and  R+(x)  by  (6),  (4),  and  (5),  respeetively.  Then  for  all 
X  G  IR^ 

W{x)  =  V~{x)  =  R+(x). 
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4  Proof  of  the  Representation 

In  this  section  we  give  the  proof  of  Theorem  3.3.  The  proof  that  the  differen¬ 
tial  game  has  value  (i.e.,  that  V~{x)  =  is  deferred  to  the  appendix. 

We  first  prove  some  preparatory  lemmas.  Let 

p{x,  a)  =  {p  ^  S  ■.  X  +  Wa{x)F{p,  a)  £  C}  . 

Lemma  4.1  Assume  Condition  3.1.  For  each  x  £  IR;^  and  a  £  A  the  set 
p{x,  a)  is  nonempty  and  convex,  and  moreover  the  mapping  from  IR:^  x  A 
to  S  defined  by  {x,  a)  p{x,  a)  is  upper  semicontinuous. 

Proof:  Fix  x  £  IR:^  and  a  £  A,  and  let  p™  come  within  1/m  of  the  infimum 
of 

inf  {a  :  X  +  o'F{p,  a)  £  C} 

over  p  £  S.  Let  cr™  come  within  1/m  of  the  infimum  over  cr  when  p  =  p™. 
By  extracting  a  subsequence,  we  can  assume  that  (p^jCr™)  ^  {p* ,Wa{x)) 
with  p*  £  S.  We  claim  that  p*  £  p{x,  a).  Indeed,  we  have 

X  +  Waix)Fip* ,a)  =  lim  (x  +  a'^Fip^ ,  a))  £  C, 

m — 

which  proves  that  p*  £  p{x,a),  and  shows  that  p(x,a)  is  nonempty.  Since 
p  ^  T(p,  a)  is  linear,  it  follows  that  p(x,  a)  is  also  convex. 

To  prove  the  upper  semicontinuity  we  first  show  that  Wa{x)  is  jointly 
continuous  in  (x,a).  Let  ^  {x,a)  i  oo.  Under  Condition 

3.1,  for  all  £  >  0  we  can  find  p  £  S  such  that  x  +  \Wa{x)  +  e\F{p,a)  £ 
C° .  This  implies  limsupj_>oo  lU„i(x*)  <  Wa{x)  +  e,  and  since  e  >  0  is 
arbitrary  limsupj^oo  lUQ,i(x*)  <  Wa{x).  Let  p*  G  p(x*,a*).  By  extracting  a 
subsequence,  we  can  assume  that  lU„i(x*)  ^  M  and  p'^  p  £  S.  Taking 
the  limit  as  i  ^  oo  in 


x*  +  lU„i(x*)F(p^a*)  G  C 


gives 

X  +  MF{p,  a)  £  C, 

and  therefore  liminfj^oo  kLQ,i(x*)  >  Wa{x).  We  conclude  that  Wa{x)  is 
jointly  continuous  in  (x,a). 

Next  let  ^  {x,a)  as  i  oo,  and  let  p*  G  p(x*,a*).  We  must 

show  that  p*  ^  p*  implies  p*  £  p{x,  a).  Using  the  continuity  of  Wa{x), 

X  +  Wa{x) F (p* ,  a)  =  lim  (x*  +  lU„i(x*)F(p*,  a*))  G  C. 

i^oo  \  / 
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We  conclude  that  p*  G  /3(x,a),  and  therefore  {x,a)  p{x,a)  is  upper 
semicontinuous.  I 


Lemma  4.2  Assume  that  Conditions  2.1  and  3.1  are  satisfied  and  define 
W(x)  for  X  G  IR'^  by  (6).  Consider  any  point  x  G  IR:^\{0}  and  let  v  G  IR'^ 
be  such  that  x  +  W{x)v  G  C.  Then 

D^W{x)  <  -1. 


Proof:  Since  x  +  W{x)v  G  C  it  follows  that  W{x  +  W{x)v)  =  0.  The 
convexity  of  W  then  implies  that  for  any  a  G  (0,  W{x)) 


Wix  +  av)  <  :^^W{x  +  Wix)v)+ (^l- ,^^'^Wix) 


<  1- 


W{x] 


W(x). 


It  follows  that 


^  .X./  X  ,  W(x  +  av)-W(x) 

D^W (x)  =  hm -  <  — 1. 

ajo  a 


Define 

B{x)  =  {F{p{x,  a),  a)  :  a  G  .A}. 

These  are  the  velocities  that  are  optimal  (for  the  true  controller)  at  x  for 
the  control  problem  Wa{x)  for  some  a  G 

Lemma  4.3  Assume  that  Conditions  2.1  and  3.1  are  satisfied  and  define 
W(x)  for  X  G  IR'^  by  (6).  Then  for  any  x  G  IR;^  and  v  G  B{x)  we  have 
X  +  W{x)v  G  C. 

Proof:  Suppose  that  v  G  F{p{x,a),a)  for  some  a  G  We  know  that 
X  +  Wa{x)v  G  C  and  ITa(x)  <  W{x). 

If  u  G  C  then  we  are  done,  since  C  is  a  cone  with  vertex  at  the  origin.  Now 
X  G  IR:^  implies  that  v  =  vi  +  V2,  where  vi  G  —  1R(J^  and  U2  G  C.  Thus 
we  need  only  show  —  IR:^  C  C.  Since  C  is  a  convex  cone,  to  show  this  it  is 
enough  to  prove  that  — e*  G  C  for  each  i  =  I, ...,  N . 
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Let  the  vectors  {d*,i  =  1, N}  be  defined  by 


1  i  =  j 
0 


Condition  2.1  implies  the  vectors  {di,  i  =  1, N}  are  linearly  independent, 
and  so  this  is  well  defined.  The  vectors  {d|,  i  =  1, ...,  N}  provide  an  external 
representation  for  C  in  that 

C  =  {yeM^  :{y,d*)<0,i  =  l,...,N}. 

Thus  — Cj  G  C  will  follow  if  we  show  {ei,d*)  >  0  for  j  =  1,  Let  D  be 

the  matrix  whose  ith  column  is  di.  Then  is  the  matrix  whose  jth  row 
is  d’j.  We  can  write  D  =  I  —  A,  where  A  is  nonnegative.  Under  Condition 
2.1  the  spectral  radius  of  A  is  less  than  one,  and  so  we  can  express  as 
This  shows  that  is  nonnegative,  and  completes  the  proof  of 
the  lemma.  I 


We  recall  the  definition  of  the  projected  velocity  given  in  (1)  and  I{x)  = 
{i  :  Xi  =  0}  for  x  G  IR;^. 

Lemma  4.4  Assume  that  Conditions  2.1  and  3.1  are  satisfied  and  define 
W(x)  for  X  G  IR'^  by  (6).  Let  x  G  IR:^  he  given.  Let  y  <  x  eomponentwise, 
and  assume  y  ^  C  (so  that  W{y)  >  0).  Let  v  G  B{y),  and  suppose  there 
exist  Oj  >  0,  f  G  / (x)  such  that 

lv+  ^  aidi,ej\=0,  j  e  L{x).  (8) 

\  / 

Let  q  =  V +  J2iei{x)  Uidi.  Then 

DqW{y)  <  -1. 

Proof:  By  Lemma  4.2  it  is  enough  to  show  that  y  +  W{y)q  G  C.  According 
to  the  last  lemma  y  +  W{y)v  G  C,  and  so  we  can  express  {y/W{y))  +  u  as 
—  didi  for  some  constants  Oi  >  0,  i  =  1, ...,  N.  To  prove 


y  +  W{y)q  = 


G 


W{y) 


{y/W{y))  +  v  +  aidi 

i€l{x) 


W{y) 

C, 


N 

^  ^  O^idi  “h  ^  ^  O^idi 
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it  is  therefore  enough  to  show  that  a*  >  a*  for  i  G  I{x). 

Since  v  =  —{y/W{y))  —  ^idi,  the  equations  (8)  can  be  rewritten  as 

i-{y/W{y))-J2didi+  aidi,ej)=0 

\  iGl{x)  / 

forj  G  /(x).  Let  M  denote  the  cardinality  of /(x).  We  recall  that  {dj,ei)  <  0 
if  i  /  j  and  <  x*  <  0  for  i  G  I{x).  As  a  consequence,  we  can  rewrite  this 
system  of  M  equations  as 

(/  -  DuV  =  q, 

where  I  is  the  M  x  M  identity  matrix,  Dm  is  non-negative  with  spectral 
radius  less  than  1,  Vj  =  aj  —  aj  for  j  G  I{x),  and  g*  =  J2j<^i{x)dj{dj,ei)  + 
{yi/W{y))  <  0  for  each  i  G  /(x).  Since  each  component  of  r  = 
is  obviously  nonpositive,  we  conclude  that  a*  <  a*  for  f  G  /(x).  ■ 

In  the  proof  of  Theorem  3.3  we  will  need  to  construct  a  nearly  optimal 
strategy  for  the  minimizing  player  to  prove  that  V~{x)  <  W{x).  If  W  were 
smooth  then  such  a  strategy  would  be  easy  to  construct.  However,  since 
W  is  only  convex  it  must  be  mollified  to  construct  this  policy,  and  this 
mollification  in  turn  complicates  the  construction  of  the  optimal  control  on 
the  boundary.  In  the  lemma  that  follows  we  apply  the  previous  lemma  to 
deal  with  this  issue. 

Lemma  4.5  Assume  that  Conditions  2.1  and  3.1  are  satisfied  and  define 
W(x)  for  X  G  IR'^  by  (6).  Let  7  >  0  6e  given.  Then  there  exists  a  eonvex, 
continuously  differentiable  and  radially  linear  function  ITy  :  IR'^  ^  [0,  00) 
such  that  for  all  x  G  IR+ ,  a  G  .4,  and  p  G  p(x,  a), 

|IT^(x)  -  IT(x)|  <  7lT(x)  (9) 

and 

(7r(x,  F{p,  a)),DW^{x))  <  -(1  -  7).  (10) 

Proof:  Fix  7  >  0.  We  begin  by  noting  a  relation  between  directional 
derivatives  and  subdifferentials  for  convex  functions.  Fix  x  G  IR+,  and  let 
dW{x)  denote  the  set  of  subdifferentials  of  W  at  x.  Then  for  any  v  G  IR'^ 
and  any  q  G  dW{x),  {q,v)  <  DyW(x).  According  to  Lemmas  4.1,  4.2, 
and  4.3,  for  each  a  ^  A  and  p  G  p(x,a)  we  have  „)IT(x)  <  —1,  and 
therefore  for  all  such  a  and  p 

(g,F(p,a))  < -I  (II) 
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for  all  q  G  dW{x). 

We  next  mollify  the  function  W .  Define  the  convex  set  G  =  {x  '.W (x)  < 
1}.  For  a  >  0  define  the  translation  Ga  =  {y  =  x  +  a(l, 1)  :  x  G  G},  and 
for  6  >  0  consider  the  (5— fattening  G^  =  {y  :  \\y  —  x||  <5  for  some  x  G  Ga}- 
Since  0  G  G°,  we  can  assume  without  loss  that  a  is  small  enough  that  the 
origin  is  contained  in  the  interior  of  As  we  will  see,  the  translation  is 
needed  to  ensure  that  the  fattening  does  not  interfere  with  the  boundary 
conditions  that  are  required  of  the  mollification.  Finally,  let 

Wa{x)  =  inf{c  >  0  :  X  G  d{cG^a)}- 
The  construction  is  illustrated  in  Figure  7. 


2 


^2 


Figure  7:  Construction  of  Gf 


It  is  easy  to  check  that  is  finite  and  convex.  Also,  it  is  well  known 
that  Ga  a-  boundary  for  each  5  >  0,  and  thus  Wf  is  continuously 
differentiable  on  IR:^\{0}.  We  first  compute  the  gradient  of  IT^.  Fix  any 
point  X  G  1R:^\{0}  and  let  n  be  the  outward  normal  to  Gf  at  y  =  x/Wa{x). 
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Since  is  radially  linear  the  gradient  of  W^(x)  must  be  proportional  to 
n,  which  means  there  must  be  a  supporting  hyperplane  of  the  form  (x,  rn) 
to  at  X  (here  we  use  the  fact  that  bba(O)  =  0).  Thus  using  the  equality 
W^{x)  =  {x,rn),  we  find  that 


DWa{x)  =  rn  = 


MMi  „  =  (^)  „ 

(i,n>  ) 


Let  y'  be  the  unique  point  in  Ga  that  is  exactly  distance  6  from  y,  and  let 
z  =  y  —  a(l, 1).  Then  n  is  also  an  outward  normal  to  G  at  z,  and  an 
analogous  calculation  to  the  one  just  given  shows  that  for  any  point  of  the 
form  bz,  b  G  (0,  oo),  (1/ {z,n))  n  is  a  subdifferential  to  W  at  bz.  Therefore 


where  g  is  a  subdifferential  to  W  at  We  can  make  \y  —  z\  as  small  as 
desired  by  choosing  a  >  0  and  6  >  0  small.  Let  p  G  p{x,a).  Since  (2/,n) 
is  uniformly  bounded  from  below  away  from  zero,  for  all  sufficiently  small 
a  >  0  and  <5  >  0  equation  (11)  implies 

(F(p,a),ZllTi(x))<-(l-7). 


Observe  that  conditions  (8)  characterize  7r(x,  F{p,  a)).  Thus  if  we  knew  that 
z  <  y  (componentwise)  then 

{'K{x,F{p,a)),DW^{x))  <  -(1-7) 


would  also  follow  from  Lemma  4.4.  However,  z  <  y  follows  easily  by  fixing 
a  >  0  and  then  choosing  6  G  (0,  a).  Finally,  it  is  also  easy  to  check  that 
G  and  Gi  can  be  made  arbitrarily  close  in  the  Hausdorff  topology,  which 
immediately  implies 

\W^{x)-W{x)\  <  -fW(x) 

when  a  and  6  are  small.  The  lemma  now  follows  by  taking  W.y  =  for 
suitable  a  >  0  and  5  >  0.  ■ 


In  the  proof  of  Theorem  3.3  we  will  use  a  verification  argument  to  show 
V~{x)  <  W^{x)  plus  a  small  error.  The  use  of  feedback  controls  for  the 
minimizing  player  would  be  problematic.  The  next  lemma  will  allow  the  use 
of  piecewise  constant  controls  and  thereby  simplify  the  proof.  The  lemma  is 
an  immediate  consequence  of  the  continuity  of  DW^{x)  for  7  >  0  and  x  7^  0. 
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Lemma  4.6  Assume  that  Conditions  2.1  and  3.1  are  satisfied  and  for^  >  0 
define  M4y(x)  for  x  G  by  Lemma  f.5.  Then  there  is  v  >  D  such  that 
for  all  z  G  IR:^  with  ||z||  =  1,  all  y  with  \\z  —  y\\  <  u,  all  a  G  A,  and  all 
p  G  p{z,a), 

{niz,  Fip,  a)),  DW^iy))  <  -(1  -  27).  (12) 


Proof  of  Theorem  3.3:  We  first  prove  that  W{x)  <  V^{x).  Fix  x  G 
®'+\  {0}  a  G  A.  Let  p  G  N  he  any  open  loop  control,  and  let  Ta,  >  0  be 
the  corresponding  first  time  that  the  origin  is  reached  by  the  solution  to 

4>{i)  =  (.4>{t),F{p{t),a)) ,  (/)(0)=x. 

If  Ta;  =  00  there  is  nothing  to  prove,  and  so  we  assume  Tx  <  00.  Using  the 
definition  of  the  Skorokhod  Problem,  there  exist  afit)  >0,1  =  l,...,N,t  G 
[0,  Tx]  such  that 

N 

<j){t)  =  F{p{t),a)  +  ai{f)di 

i=l 

for  almost  every  t  G  [0,ra;].  Integrating  over  [0,ra;]  and  using  the  definition 
p  =  ^  p(t)dt,  we  find  that 

-X  =  TxF(p,  a)  -lu 

for  some  co  G  C,  and  so  x  +  TxF{p,a)  G  C.  The  definition  of  Wa(x)  then 
implies  Tx  >  Wa{x).  Since  0[p]{t)  =  a  is  a  legitimate  strategy  to  use  in  the 
definition  of  V^{x)  and  p  G  N  \s  arbitrary,  it  follows  that  V^{x)  >  Wa{x) 
for  all  a  G  A.  Taking  the  supremum  on  a  G  ^  gives  V^{x)  >  W{x). 

We  next  prove  W{x)  >  V~{x).  Let  7  G  (0, 1/2),  and  let  >  0  be  given 
according  to  Lemma  4.6.  Fix  x  G  IR+\  {0}  and  let  the  open  loop  control 
a  G  M  be  given.  We  recursively  construct  a  strategy  5  G  A  as  follows.  Given 
a  point  of  the  form  x*  7^  0  (with  xq  =  x)  and  corresponding  times  Ti  (with 
To  =  0),  we  consider  the  normalized  version  Zi  =  Xj/||xj||.  Let  p*{x,a)  be 
any  single- valued  and  measurable  selection  from  p{x,a).  We  define  5[a](t) 
for  t  G  [Tj,Ti+i)  to  be  p*{zi,a{t)),  where  Tj+i  >  Ti  is  defined  by 

inf{t  >  Ti  :  ||(/(t)/||(/(t)||  -  Zi\\  >iz}L  inf{t  >  Ti  :  ^{t)  =  0}, 


where 

fi{t)  =  T:{fi{t),F{p*{zi,a{t)),a{t))),  (/(xj)  =  Xj. 


Since  the  speed  ||</>(t)||  is  uniformly  bounded  from  above,  it  is  easy  to  check 
that  inf{t  >  Tj  :  ||</'(i)/||</'(t)||  —  Zi\\  >  v}  —  Ti  is  uniformly  bounded  away 
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from  zero  if  xi  is  in  a  closed  set  that  does  not  contain  the  origin.  We  will 
make  use  of  the  fact  that  for  any  x  /  0  and  any  v  n{x,v)  =  7r(x/||x||, n). 
According  to  Lemma  4.6, 

=  {DW^{(l){t)),7T{4i{t),  F{p*{zi,a{t)),a{t))))  <  -I  +  27 

for  almost  every  t  prior  to  the  first  time  (j)  hits  the  origin,  and  therefore  for 
all  such  times 

-  W^{x)  =  f  {DW^{(j){t)),<j){t))ds  < -t{l -2j). 

Jo 

We  conclude  that 

W^m))<W^{x)-t{l-2j), 

and  therefore  (j)  reaches  the  origin  by  time  lWy(x)/(l  —  27).  This  implies 
<  lAiY(x)/(l  —  27),  and  since  7  >  0  is  arbitrary,  that  V~{x)  <  W{x). 
Thus  we  have  shown  that  V~{x)  <  W{x)  <  V~^{x).  The  proof  that 
V~{x)  =  V^{x)  is  based  on  a  uniqueness  result  for  the  corresponding  PDE, 
and  is  presented  in  the  Appendix.  This  completes  the  proof  of  the  theorem. 


5  Synthesis  of  Controls 

The  “true”  controls  used  to  prove  W (x)  >  V~  (x)  in  the  proof  of  Theorem 
3.3  are  not  very  useful,  since  they  require  knowledge  of  the  control  that 
nature  applies  at  all  times.  In  this  section  we  will  formally  discuss  how  to 
construct  controls  that  are  optimal  (or  nearly  optimal),  and  which  depend 
only  on  the  state  of  the  network.  A  rigorous  proof  will  appear  elsewhere. 
Formally,  W  =  is  the  solution  to  the  equation 

sup  inf  [{DW{x),  F{p,a))  +  1]  =  0,  x  G  (1R:^)°,  (13) 

aeA 

together  with  the  boundary  conditions 

{DW{x),di)  =  0,  ie  /(x), X  G  aiR^\{0},  IT(0)  =  0.  (14) 

Since  F  is  affine  in  each  variable  seperately  and  A  and  S  are  compact  and 
convex,  [19,  Corollary  37.6.2]  implies  that  the  sup  and  inf  in  (13)  can  be 
interchanged  (i.e.,  one  expects  the  game  to  have  value). 

Since  W  is  not  necessarily  smooth  we  cannot  expect  a  classical  sense 
solution  to  (13)-(14),  and  so  one  must  consider  a  weak  sense  solution,  e.g.. 
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viscosity  solutions.  Because  W  is  convex,  the  set  of  subdifferentials  to  W  at 
X  (denoted  D^W{x))  is  never  empty.  It  follows  from  the  characterization 
of  viscosity  solntions  (see  the  Appendix)  that  for  any  q  G  D^W{x)  there 
exists  at  least  one  saddle  point  {p{q),a{q))  such  that 

sup {q,F{p{q),  a))  <  -1. 

aGA 

Let  R(q)  denote  the  set  of  all  points  p  £  S  which  have  this  property.  It  is 
easy  to  check  that  this  set- valued  function  is  upper  semicontinuous:  qn  ^  Q, 
pn  ^  P  and  pn  £  R{qn)  implies  p  £  R{q).  At  each  point  x  £  IR;^  we  define 
a  set  of  controls  S{x)  C  5  by 

S{x)  =  ^q(zD-W{x)R{<l)- 

Note  that  since  x  D^W{x)  and  q  R{q)  are  upper  semicontinuous,  so 
is  the  composition  S{x),  and  that  the  radial  linearity  of  W  implies  a  radial 
homogeneity  of  S:  S{ax)  =  S{x)  for  all  x  £  IR:^  and  a  £  (0,oo).  The  set  of 
conjectured  controls  for  x  in  the  interior  is  then  S{x). 

However,  when  on  the  boundary  we  must  be  more  careful.  As  can  easily 
be  seen  by  considering  two  dimensional  examples,  there  is  an  important 
distinction  depending  in  whether  the  boundary  condition  holds  in  a  classical 
sense  or  not.  The  following  conjectures  for  the  form  of  the  optimal  control 
are  based  on  the  analysis  of  two  dimensional  examples,  and  have  not  been 
verified  in  any  generality.  Let  us  first  consider  the  case  of  a  point  x  where 
I{x)  =  i  for  a  single  value  i.  In  this  case  the  classical  sense  formulation  of 
the  boundary  condition  is  {DW {x),di)  =  0.  If  this  condition  holds,  it  means 
that  all  optimally  controlled  trajectories  push  into  the  boundary,  and  that 
any  selection  from  S{x)  is  optimal.  If  however  {DW{x),  di)  ^  0,  then  even  if 
some  elements  from  S{x)  lead  to  trajectories  that  push  into  the  boundary,  we 
must  restrict  ourselves  to  only  those  for  which  the  saddle  point  dynamics 
do  not  push  strictly  into  the  boundary.  If  the  boundary  condition  is  not 
valid  in  the  classical  sense,  then  we  conjecture  that  this  set  is  always  non¬ 
empty.  Analogous  considerations  hold  for  the  points  at  the  intersection  of 
two  or  more  faces.  In  general,  choosing  a  control  for  which  the  saddle  point 
dynamics  push  into  a  face  is  only  allowed  when  the  corresponding  boundary 
condition  holds  in  the  classical  sense. 

6  Appendix 

In  this  appendix  we  will  prove  that  the  game  has  valne,  i.e.,  that  V^{x)  = 
V~{x)  for  all  X  £  IR:^.  A  key  ingredient  is  a  uniqueness  result  for  the  partial 
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differential  equation  (PDE)  that  and  should  satisfy.  An  excellent 
general  reference  for  the  theory  of  viscosity  solutions  of  first  order  nonlinear 
PDE  is  the  book  [2],  The  particular  results  we  will  need  can  be  found  in  [1] 
(see  also  [9]). 

For  q  G  define 

H{q)  =  maxmin  [(g,  F(/9,  a))  +  1] 
ogT  pg5 

=  minmax  [(g,  F(/3,  a))  +  1] , 
pg5  aeA 


where  the  two  expressions  on  the  right  hand  side  are  equal  since  F(p,  a)  is 
affine  in  each  variable  separately  and  S  and  A  are  convex  and  compact.  Con¬ 
sider  a  Lipschitz  continuous  function  V  :  IR:^  ^  IR,  and  for  a  continuously 
differentiable  function  g  :  IR'^  ^  IR  let  y  be  a  local  maximum  (respectively. 

minimum)  of 

X  ^  V (x)  —  g(x). 

Then  V  is  called  a  viscosity  subsolution  (respectively, 
tion)  to  (13)  and  (14)  if 

viscosity  supersolu- 

H(Dg(y))  V  max  (^^(y),  dj)  >  0 
*6/(2/) 

(15) 

(  iL(Dy(y))  A  min  (Dy(y),  dj)  <  0  ) 

,  (16) 

and 

R(0)  <  0,  (R(0)  >  0). 

(17) 

We  henceforth  drop  the  adjective  “viscosity,”  and  note  that  a  function  that 
is  both  a  sub  and  supersolution  is  called  a  solution. 

Recall  that  V  :  IR;^  ^  IR  is  said  to  be  radially  linear  if  V (ax)  =  aV (x)  for 
all  X  G  IR:^  and  a  G  [0,  oo).  According  to  [1,  Theorem  4.3],  there  is  only  one 
function  V  satisfying  the  following  conditions:  (i)  R  is  a  viscosity  solution 
to  (15)-(17),  (ii)  V  is  Lipschitz  continuous  and  radially  linear,  and  (iii) 
R(x)  >  0  for  X  G  (IR:^)\{0}.  Suppose  that  R+  and  satisfy  conditions  (ii) 
and  (iii)  of  the  last  sentence.  Then  standard  arguments  based  on  dynamic 
programming  can  be  used  to  show  that  (i)  holds  ([1,  Theorem  3.2]  and  [2, 
Chapter  VIII]).  Thus  will  follow  if  we  can  prove  that  (ii)  and  (iii) 

hold  for  both  and  V^. 

Assume  for  now  that  R+  is  uniformly  bounded  on  bounded  sets.  It 
follows  from  Theorem  3.3  that  0  <  V~(x)  <  V'^(x)  <  oo.  It  is  also  imme¬ 
diate  from  the  definitions  that  both  V^{x)  and  V~{x)  are  radially  linear. 
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and  that  V^{x)  A  V~{x)  >  0  for  x  G  (]R:^)\{0}.  Thus  if  is  uniformly 
bounded  on  bounded  sets,  all  that  needs  to  be  shown  is  that  and 
are  Lipschitz  continuous.  We  give  the  proof  for  V^,  and  note  that  the 
proof  for  is  analogous.  Let  M  =  maxj^,||j^ll=i  y+(y),  and  assume  for 
now  that  M  <  oo.  Owing  to  the  radial  linearity,  V^{x)  <  M||x||.  Fix 
points  x,y  G  IR:^  and  e  >  0.  Let  iL  <  oo  be  the  Lipschitz  constant  of 
the  Skorokhod  Map  defined  in  Section  2.  We  claim  that  is  Lipschitz 

continuous  with  constant  MK.  The  proof  adapts  a  standard  argument  [2]. 
Choose  0  G  ©  such  that  V'^{x)  <  Cx{p-,0[p\)  +e/2.  Since  9  is  sub- 

optimal  at  y,  Cy{p,9\p]),  and  hence  there  is  p  such  that 

y^{y)  >  Cy{PiG[p])  ~  s/2-  Note  that  also  C'^(x)  <  Cx{p-,9\p])  +  e/2,  and 
hence 

C+(x)  -  V+{y)  <  Cx{pAp\)  -  Cy{pAp\)+e. 

If  Cy{p,  9[p])  >  Cx{p,  d[p])  (i-e.,  it  takes  longer  to  reach  the  origin  from  y  than 
x)  then  of  course  V~^{x)  —  V^{y)  <  e.  On  the  other  hand,  if  Cy{p,9[p])  < 
Cx{p,  9[p\)  then  we  can  stop  the  process  that  was  started  at  x  at  time  a  = 
Cy{p,9[p\).  If  we  let  (lf{t)  and  denote  the  processes  started  at  the 

points  X  and  y,  then  the  Lipschitz  property  of  the  Skorokhod  Map  implies 
||(/)^((j)  —  </^((t)||  <  K\\x  —  y\\.  Since  =  0,  this  means  that  ||</^(o')||  < 

iL||x  —  y||.  We  can  now  use  dynamic  programming  to  argue  that  IA+(x)  < 
a  +  V^{(l)^{a))  +  e/2  <  a  +  MK\\x  —  y\\  +  e/2,  and  thus  V~^{x)  —  V^{y)  < 
MK\\x  —  y\\  +  e.  Combining  the  two  cases  and  using  that  e  >  0  is  arbitrary, 
it  follows  that  C+(x)  —  V^{y)  <  MK\\x  —  y\\  for  all  x,y  G  IR;^. 

With  the  proof  that  and  are  Lipschitz  continuous  complete,  all 
that  remains  is  to  prove  that  is  uniformly  bounded  on  bounded  sets. 
Under  Condition  2.1,  it  was  shown  in  [8,  Lemma  2.1,  Theorem  2.1,  and 
page  60]  that  there  is  a  compact,  convex  set  B  C  IR'^  with  the  following 
properties: 


LOG 

2.  if  z  ^  dB  and  n  is  an  outward  normal  to  B  at  z,  then  \{z,ei)\  <  1 
implies  (n,  di)  =  0, 

3.  if  z  G  dB  and  n  is  an  outward  normal  to  B  at  z,  then  {z,  ei){n,  di)  >  0. 


By  considering  sets  of  the  form  B^  =  {y  :  \\y  —  xjj  <6  for  some  x  G  B} 
(with  6  >  0),  it  is  easy  to  verify  that  without  loss  we  can  assume  B  has  a 
continuously  differentiable  boundary.  Define  the  function  R  :  IR'^  ^  [0, 
by 

R{x)  =  inf{c  :  x  G  d{cB)}. 


,  oo 
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^From  the  convexity,  properties  1  and  2  listed  above,  and  the  smoothness  of 
dB,  it  follows  that  R  is  continuously  differentiable  save  at  x  =  0,  and  that 
for  X  G  IR:^\{0} 

{DR{x),di)  =  0  if  i  G  I{x).  (18) 

Now  fix  any  point  x  G  IR:^\{0},  and  let  z  G  dB  satisfy  z  =  ax  for 
some  a  G  (0,  oo).  If  n  is  the  corresponding  outward  normal  to  B  at  z,  then 
DR{x)  =  bn  for  some  b  G  (0,oo).  According  to  properties  2  and  3  above, 
{DR{x),di)  >  0.  Let  the  vectors  d*,i  =  be  defined  by  {di,d*)  =  6ij, 

where  6ij  is  1  if  i  =  j  and  0  otherwise.  These  vectors  are  well  defined, 
since  Condition  2.1  implies  the  linear  independence  of  {di,i  = 

Writing  DR{x)  =  Cidl,  it  follows  from  {DR{x),  di)  >  0  that  Q  >  0  for 
i  =  We  now  apply  Condition  3.1.  It  follows  from  this  condition 

that  for  each  a  G  A  there  is  p  G  S  and  c  >  0  such  that 

{DR{x),  F{p,a))  <  —c. 

Since  A  is  compact,  an  open  covering  argument  shows  the  existence  of  c  >  0 
such  that 

max'niin{DR{x),  F{p,a))  <  —c.  (19) 

a&A  p&S 

Finally,  the  radial  linearity  of  R,  the  continuity  of  DR{x),  and  another  open 
covering  argument  that  uses  the  compactness  of  dB  n  IR:^  shows  that  c  >  0 
can  be  selected  so  that  (19)  holds  for  all  x  G  ]R:^\{0}. 

Equations  (18)  and  (19)  imply  that  R/c  is  a  (classical)  supersolution  to 
(13)  and  (14).  Standard  arguments  based  on  dynamic  programming  can 
then  be  used  to  show  that  V^{x)  <  R{x)/c.  (See,  for  example,  the  proof 
of  Theorem  3.3.)  This  completes  the  proof  that  V^{x)  =  V~{x)  for  all 
X  G  IR^. 
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